{/* banner prompt: A split diptych, flat editorial illustration, 16:9. Left: a gleaming surgical scalpel labelled "reasoning" clumsily trying to paint a watercolour, ink smeared and blotchy. Right: a calm seated person writing clean elegant script on a sheet of paper, holding a single pen in one hand — anatomically correct, exactly two arms and two hands, only one pen, the other hand resting flat on the paper. Clean muted editorial palette, minimal, conceptual. --ar 16:9 */}

Last time, the [4 GB card you already own](/blog/2026/06/17-the-4-gb-card-you-already-own-can-reason-now/) turned out to reason like a giant: **VibeThinker-3B**, a 3-billion-parameter specialist, solved competition math that models twice its size could not — entirely on a modest laptop GPU. That was a genuinely surprising result, and it left an obvious, almost lazy follow-up question hanging in the air.

If a 3B model on a gaming laptop is clever enough to crack an olympiad problem, surely it can knock out a short story or a tweet?

No. It cannot. And the way it fails is far more interesting than a simple "it's not very good," because it tells you something true about *all* of these small models — and about which one you should actually reach for when you want words instead of answers.

## The test it should have passed

Same laptop, same spirit as the rest of the series: give the model a job and judge the result honestly. The jobs here are the easiest imaginable — the sort of thing you would dash off without thinking:

- Write a **120-word** short story about a lighthouse keeper who finds a message in a bottle.
- Write a warm, **4-sentence** email politely declining a meeting and proposing to handle it async.
- Write a **4-line** poem about autumn rain.

VibeThinker got the poem. On the other two, it produced **nothing at all** — not a bad story, not a wrong story, *no story*. It ran for 42 seconds, generated over eight thousand characters, and never once emitted the piece it was asked for.

Where did all that effort go? Into **counting**. Here is the shape of what it did, lightly trimmed:

```text
We must generate exactly 120 words, counting each word separated by spaces...
First draft: "The lighthouse keeper, Elara, tended her beacon each midnight.
One stormy night a wooden bottle appeared on the rocks..."
Now count words. Let's count manually:
The(1) lighthouse(2) keeper,(3) Elara,(4) tended(5) her(6) beacon(7) each(8)
midnight.(9) One(10) stormy(11) night(12) a(13) wooden(14) bottle(15)...
```

It wrote a perfectly serviceable draft — "its beam cutting through the fog like a whispered hope" is a nice line — and then buried it under an obsessive, self-appointed audit of the word count, recounting from the top, second-guessing whether punctuation counts, and running clean out of budget mid-tally. The 4-sentence email died the same way, deliberating about whether a greeting counts as a sentence until the tokens ran out.

The tell is what happens when you remove the number. Ask it for "a short atmospheric story about a lighthouse keeper" — no count — and it finishes in 16 seconds with a genuinely lovely 385-word piece. So the prose is not the problem. **The constraint is.**

## Why the training that won the math lost the tweet

This is the whole point, so it is worth saying plainly. VibeThinker is trained for *verifiable* reasoning — problems where an answer can be mechanically checked. That training instilled one deep reflex: **when there is a checkable target, work it relentlessly until it is provably satisfied.** On an AIME problem, that reflex is a superpower.

Now hand that same reflex a writing task with a number in it. It does not see "write me something nice, roughly this long." It sees a *verifiable target* — exactly 120 words — and does the only thing it knows how to do with a verifiable target: reason about it, exhaustively, checking and rechecking. It optimises the one trivial, checkable part of the request and never delivers the part you actually wanted. The scalpel that so precisely dissects a maths problem is trying to paint a watercolour, and it keeps stopping to measure the brush.

Its own model card warned us, in so many words: this is a specialist; for open-domain work, use a general model. Last post that was an abstract caveat. This is what it looks like in practice — the model is not *bad at prose*, it is *mis-aimed at prose*.

## So who can write? The boring models

The fix is not a cleverer model. It is a *less* clever one — a plain **instruct** model with no reasoning theatrics. I pulled two that fit the 4 GB card easily and ran the identical battery:

- **`gemma2:2b`** — Google's Gemma 2, 2 billion parameters, ~2 GB on disk.
- **`qwen2.5:3b`** — Qwen2.5 3B *Instruct*, the general-purpose sibling of the `qwen2.5-coder:3b` we use for code.

<Table data={
    {
        headers: ['Task', 'VibeThinker-3B', 'gemma2:2b', 'qwen2.5:3b'],
        rows: [
            ['120-word story', 'never finished (counting loop, 42 s)', 'done, ~8 s', 'done, 7 s'],
            ['4-sentence email', 'never finished', 'done, 1.4 s', 'done, 1.4 s'],
            ['4-line poem', 'done', 'done, 0.9 s', 'done, 0.7 s'],
            ['Reasoning overhead', 'always (long think)', 'none', 'none'],
        ]
    }
} />

Both instruct models delivered every piece, in seconds, with no thinking detour at all. The reasoning model spent 42 seconds failing to produce a story; the 2B model produced one in eight. And the prose was not just present — it was good. Gemma's lighthouse story opened with "The salt spray stung Thomas's weathered face" and turned on a real hook — *"Find the siren's song, she holds the key."* For a 2-billion-parameter model that fits in the change drawer of your VRAM, that is startling.

Across the battery, **`gemma2:2b` wrote the nicest English prose** — the most vivid story, the most natural-sounding email, the best poem imagery — and it did it from the *smallest* footprint of any model in this series. Google clearly tuned its little models hard for exactly this. `qwen2.5:3b` was a close, dependable second and the better all-rounder if the model also has to answer questions or reason a little.

One honest caveat that applies to *all* of these small models, not just the reasoning one: **none of them nails an exact count.** Gemma's "120-word" story came back at 136 words; Qwen's at 95. The difference between them and VibeThinker is not precision — it is that they hand you a usable draft to trim, instead of an empty tally. The practical rule writes itself: never give a small model an exact number. Ask for "around 120 words" and trim it yourself.

## The language twist

Gemma wins English. But there is a catch that matters if you are not writing in English, and it flips the ranking.

Gemma 2 is English-first. Ask it for a **Chinese** tweet and it produces something fluent and grammatically correct — but subtly *translated*, reading like polished textbook Chinese rather than something a person would actually post. `qwen2.5:3b`, from Alibaba and trained heavily on Chinese, reads like a native did write it. Asked for a relaxed weekend-coffee tweet, it reached for real internet register — *窝在家里* (curl up at home), the trendy clipped *暖咖* for a warm coffee, a playful hashtag, even the native *～* trailing tone. For a shop opening it opened with the correct idiom, *开业大吉*. A tech tweet called an old laptop *老本子* — exactly the slang a Chinese tech user would type.

The lesson is not "Qwen is better." It is that **"good at writing" is not a single axis.** Gemma is the better English stylist; Qwen is the better Chinese one — because writing quality is downstream of what a model was trained on, and these two were raised on different languages. The same trap as the reasoning model, one level up: match the tool to the *job*, and here the job includes the language.

## Match the model to the job

Step back and the whole small-model series keeps arriving at the same place. Capability is not one number you can rank models by; it is a *profile*. A 4 GB laptop can now cover an astonishing spread of work — but not with one model. With several, each a specialist, and not one of them the "smartest" in the room:

<Table data={
    {
        headers: ['The job', 'The model to reach for'],
        rows: [
            ['English prose and creative writing', 'gemma2:2b'],
            ['Chinese writing, and general all-round text', 'qwen2.5:3b'],
            ['Writing code', 'qwen2.5-coder:3b'],
            ['Hard, verifiable math and reasoning', 'VibeThinker-3B'],
            ['Driving an agent via native tool calls', 'qwen3:1.7b'],
        ]
    }
} />

VibeThinker earns its place on that list — for its row. It is a scalpel, and a scalpel is the wrong thing to write a letter with, not a worse thing than a pen. The mistake is expecting the model that aced the maths olympiad to also be the one you hand a tweet. It is the [dispatch lesson](/blog/2026/06/15-the-small-models-real-job-is-dispatch-not-work/) again, in a new suit: stop asking "which small model is best," and start asking "best at *what*."

## Reproduce it yourself

Both writers are one command each, and both sit comfortably on a 4 GB GPU:

```sh
ollama pull gemma2:2b     # ~2 GB — best English prose
ollama pull qwen2.5:3b    # ~2 GB — best Chinese, best all-rounder
```

Then hand one a writing task — and mind the trap that broke the reasoning model:

```sh
# Don't do this to a reasoning model — the exact count sends it counting forever:
#   "Write a 120-word story ..."
# Do this instead, with any instruct model:
curl -s http://localhost:11434/api/chat -d '{
  "model": "gemma2:2b",
  "messages": [{"role":"user","content":"Write a short atmospheric story about a lighthouse keeper who finds a message in a bottle."}],
  "stream": false
}' | python3 -c 'import json,sys; print(json.load(sys.stdin)["message"]["content"])'
```

If you *do* want to feel the failure for yourself, run the same 120-word prompt against `vibethinker3b` and watch it disappear into an arithmetic trance. It is a strangely instructive thing to see: a model too smart, in exactly the wrong way, for the simplest job on the list.

*The laptop now holds four little specialists — a coder, a reasoner, an English writer, a Chinese writer — none of them a generalist, all of them fast and free. Which raises the question this series keeps circling back to: can a small model be trusted to pick the right specialist for an incoming request on its own, and route to it, without a human deciding first?*


4843de1b848b2ee1787997bdc4cd3e2abe7d82c4e08d3925966d98b422c82476

Testing small local models on writing: VibeThinker-3B's reasoning training makes it fail constrained prose, while gemma2:2b (English) and qwen2.5:3b (Chinese) write cleanly on a 4 GB GPU. The specialist-vs-generalist lesson, told in prose.

The 4 GB series found a 3B that reasons like a giant. So surely it can write? It can't — and how it fails is the interesting part. VibeThinker-3B chokes on a 120-word story while two plain instruct models a fraction as clever nail it, and a language twist decides which one you want.

The Model That Aced Olympiad Math Can't Write a Tweet

{/* banner prompt: A detailed architectural blueprint diagram of a laptop, where the 4GB GPU is highlighted with glowing circuit-like patterns, and annotated with technical notes like '3B Specialist Core' and 'Low Power Draw,' showing a complex logical path that resolves entirely within the chip. --ar 16:9

Additional instructions: a little giant */}

The [dispatch experiment](/blog/2026/06/15-the-small-models-real-job-is-dispatch-not-work/) ended on a clean idea and a loose end. The idea: a small local model earns its keep not by *doing* the work but by *routing* it — a 1.7B workhorse covers about 90% of a general workload, and you escalate the hard ~10% tail to something more capable. The loose end: escalate to **what?** "More capable" almost always means reaching past the laptop for a cloud API — which means a bill, a network round-trip, and your prompts leaving the building.

This post closes that loop with a model that landed in the middle of that open question and that the internet spent a week arguing about: **VibeThinker-3B.** And it does it on the cheapest hardware in the series so far.

## You already own the hardware

Here is the part worth saying plainly. The machine in these experiments is a **gaming laptop with an RTX 3050 and 4 GB of VRAM** — not a workstation, not a rented H100, a laptop you might have bought for games years ago. **Most of us already have a card roughly like this**, gathering dust under a pile of Chrome tabs. The interesting question is not "what can a $30,000 server do." It is: *what can the thing you already own do, right now, for free?*

And there is a second half to that question that people skip: **what does it cost to run?** I sampled the GPU's power draw during inference. Idle it sits near **14 W**; flat-out solving a hard problem it averages about **50 W and peaks at 60 W**. That is a bright lightbulb — not a space heater, and an order of magnitude under a desktop 4090 (~450 W) or a datacenter accelerator (~700 W). A hard answer that takes ~50 W for ~20 seconds is about **0.3 watt-hours**; you could run a thousand of them for roughly the cost of a cup of coffee. You do not need to worry about your electricity bill. That changes the calculus of "just keep it running locally."

So: a card you already own, drawing a lightbulb of power. The only thing missing was a model worth pointing it at.

## What VibeThinker-3B is

VibeThinker-3B is a **3-billion-parameter reasoning model** from WeiboAI (Weibo's AI lab), and — conveniently for this series — it is a **fine-tune of the very same Qwen2.5-3B base** we have been running all along. That makes it an unusually fair comparison: same architecture, same footprint, different training.

The claims are what made it go viral: **94.3 on AIME 2026**, **80.2 Pass@1 on LiveCodeBench v6**, and parity with much larger frontier reasoning models on verifiable benchmarks — from a model small enough to fit in 4 GB. Predictably, [the benchmarks got argued over](https://venturebeat.com/technology/why-weibos-tiny-vibethinker-3b-has-the-ai-world-arguing-over-benchmarks-again) (generous pass@k budgets, among other things), so the honest move is the one this series always makes: stop reading the leaderboard and **run it ourselves.**

One caveat up front, straight from the model card: this is a **specialist.** It is trained for *verifiable* reasoning — competition math, coding, STEM — where an answer can be checked. Its own authors say that for broad open-domain knowledge you still want a general model. That caveat is not a footnote; it is exactly why this model fits the *escalation* slot and not the *workhorse* slot.

## The test

Same laptop, same spirit as the [4 GB experiment](/blog/2026/06/14-the-4-gb-experiment-what-actually-makes-a-small-model-a-good-coding-agent/): a hidden, automated grader is the only judge. I pulled the Q4_K_M GGUF, applied the paper's recommended sampling (temperature 1.0, top-p 0.95), and ran a four-task battery chosen so every answer is mechanically checkable — two competition-math problems, one genuinely hard [AIME](https://en.wikipedia.org/wiki/American_Invitational_Mathematics_Examination)-level problem, and a classic LeetCode coding task:

<Table data={
    {
        headers: ['Task', 'Expected', 'VibeThinker-3B', 'Time'],
        rows: [
            ['Remainder of 7^100 mod 13', '9', 'correct', '13 s'],
            ['Tilings of a 2x10 board by dominoes', '89', 'correct', '20 s'],
            ['AIME: count triples a+b+c=300 with a constraint', '601', 'correct *', '247 s'],
            ['longest_palindromic_substring in O(n^2)', '(verified)', 'correct', '16 s'],
        ]
    }
} />

Four for four. The code I ran against a brute-force oracle on a batch of strings — it matched everywhere. And the whole time the model sat **100% on the GPU**, using 2.3–2.7 GB of the 4 GB, generating at **50–63 tokens per second.** No CPU offload, no thrashing. The asterisk on the AIME row is the one real gotcha, and it is worth a section of its own.

## The gotcha: token budget, not VRAM, is the wall

On my first AIME attempt the model **failed** — but not because it could not solve the problem. With a 16K-token context it generated **16,299 tokens of reasoning** and ran straight into the context wall mid-thought, never reaching an answer. Reasoning models think in the open, and a hard problem is a long think.

I raised the context to 32K and re-ran the exact same prompt. This time it converged — **601, correct** — in about 12,000 tokens. And here is the lovely part: even at 32K context it *still* sat entirely on the GPU, at 2.7 GB.

That is the counter-intuitive lesson for this class of model on small hardware: **the binding constraint is the token budget, not the VRAM.** The KV cache (quantized, with flash attention) is cheap; what these models need is *room to think*. If you run a reasoning model on a 4 GB card and it "fails," check whether it actually finished before you blame the model.

## The head-to-head

Solving four problems is nice, but the real question is comparative: against the models we already run on this laptop, is VibeThinker actually *better*, or just *different*? I ran the identical battery against `qwen2.5-coder:3b` (our coding workhorse, same 3B size) and `qwen3:4b` (the reasoning-tuned model one size up).

<Table data={
    {
        headers: ['Task (expected)', 'VibeThinker-3B', 'qwen2.5-coder:3b', 'qwen3:4b'],
        rows: [
            ['7^100 mod 13 (9)', 'correct, 13 s', 'correct, 11 s', 'correct, 140 s'],
            ['2x10 dominoes (89)', 'correct, 20 s', 'wrong (got 11)', 'correct, 595 s'],
            ['AIME triples (601)', 'correct, 247 s', 'wrong (got 3)', 'failed, ran 2.2 hours'],
            ['palindrome O(n^2)', 'correct, 16 s', 'correct, 9 s', 'correct, 304 s'],
            ['Score', '4 / 4', '2 / 4', '3 / 4'],
            ['Speed', '50-63 tok/s', '~65 tok/s', '3-18 tok/s'],
            ['GPU placement', '100% GPU', '100% GPU', 'spills to CPU'],
        ]
    }
} />

Read that table for a minute, because each column tells a different story.

**`qwen2.5-coder:3b`** is the fastest thing here and stays fully on the GPU — but it has no reasoning chain, so it answers from the hip and falls apart on anything non-trivial. It missed the dominoes problem (a recurrence a reasoning model finds in seconds) and was nowhere near the AIME answer. Superb at code, useless at hard math. Exactly the profile that made it our *editor* in the earlier studies, and exactly why it cannot be the escalation target.

**`qwen3:4b`** *can* reason — notice it got the dominoes problem the coder missed. But it is 4B, and 4B does not fit in 4 GB. It spills onto the CPU and collapses to **3–18 tokens per second.** The AIME problem ran for **two hours and twelve minutes** and *still* did not finish inside a generous token cap. It is the right kind of model at the wrong size for this card — technically capable, practically unusable here.

**VibeThinker-3B** is the only column that is both *right* and *runnable*. It reasons like the 4B but fits and flies like the 3B. That is the whole point: it gives you **qwen3:4b's reasoning at qwen2.5-coder:3b's speed and footprint** — because it *is* the 3B, just trained to think.

## The payoff: the tail finally has a target

Bring it back to the [dispatch thesis](/blog/2026/06/15-the-small-models-real-job-is-dispatch-not-work/). That study argued you should route by **measured capability**, not by parameter count: let a small workhorse handle the bulk, and escalate the hard tail to whatever model actually clears it. The unsatisfying gap was that "whatever clears it" pointed off the laptop and into the cloud.

VibeThinker-3B fills that gap for one specific, valuable slice of the tail: **hard, verifiable reasoning.** A math or coding problem your 1.7B workhorse flubs and your 4B can't run in time, this 3B specialist solves — locally, on the GPU you already own, for a lightbulb of electricity. You escalate *sideways into a specialist* instead of *up into the cloud.*

It is not a general assistant. Ask it for open-domain knowledge or to drive an agent's tool calls and it will lose to qwen3 — it is a scalpel, not a Swiss Army knife. But a scalpel is exactly what an escalation slot wants. The dispatch post said the small model's job is to know *who* should do the work. This is one more capable "who" — and remarkably, it is one that still fits in 4 GB.

## Reproduce it yourself

The whole point of this series is that you can run it on the machine you already have. Here is the entire setup.

Pull the quantized model straight from Hugging Face into [Ollama](https://ollama.com/) (no separate download step needed):

```sh
ollama pull hf.co/mradermacher/VibeThinker-3B-GGUF:Q4_K_M
```

Wrap it in a Modelfile so the recommended sampling and a generous context are baked in. The 32K context is the important part — give the reasoner room to think:

```dockerfile
FROM hf.co/mradermacher/VibeThinker-3B-GGUF:Q4_K_M
PARAMETER temperature 1.0
PARAMETER top_p 0.95
PARAMETER top_k 0
PARAMETER num_ctx 32768
```

```sh
ollama create vibethinker3b -f VibeThinker.Modelfile
```

Then point a problem at it and confirm it lands fully on the GPU:

```sh
curl -s http://localhost:11434/api/generate -d '{
  "model": "vibethinker3b",
  "prompt": "In how many ways can a 2x12 rectangle be tiled by 1x2 dominoes? End with the final integer on its own line.",
  "stream": false
}' | python3 -c 'import json,sys; print(json.load(sys.stdin)["response"])'

ollama ps   # PROCESSOR should read 100% GPU
```

If a hard problem comes back without a final answer, you have hit the token wall, not a reasoning failure — raise `num_ctx` and run it again.

*The series has gone from reading agents to running them, and from "what fits in 4 GB" to "what's worth pointing 4 GB at." Next, the obvious question once you have a workhorse and a specialist on the same laptop: can a small model reliably decide, on its own, which of the two a given problem needs — and hand off without a human in the loop?*


b7a6e0bfc0313d9f1f548829fc58f29797d25c1596f8e8501df822d3706e2e91

VibeThinker-3B tested on a 4 GB RTX 3050: a 3B verifiable-reasoning specialist that fits fully on a modest laptop GPU, beats qwen2.5-coder and qwen3:4b on hard math at a fraction of the power, and fills the escalation slot from the small-model routing study.

The dispatch experiment ended on a question: route the hard tail to a more capable model — but which one, if you do not want to reach for the cloud? VibeThinker-3B, a 3B reasoning specialist, runs entirely on the 4 GB laptop GPU you probably already own, solves competition math the workhorse cannot, and sips about 50 watts doing it.

The 4 GB Card You Already Own Can Reason Now

{/* banner prompt: High-key cinematic shot in a vast minimalist white space with soft haze; a single small luminous cube acts as a switchboard, sorting incoming streams of light — most loop neatly back into the cube itself, while one thin, bright tributary branches off toward a distant towering machine that glows on the horizon; 16:9, volumetric lighting, a sense of triage, flow, and economy. */}

The [4 GB experiment](/blog/2026/06/14-the-4-gb-experiment-what-actually-makes-a-small-model-a-good-coding-agent/) ended on an honest wall: a small local model fits in a 4 GB laptop GPU and solves easy coding tasks, but it hits a ceiling on hard ones and [cannot drive a native tool call](/blog/2026/06/15-what-a-native-tool-call-actually-is/) at all. The natural reaction is to shrug and reach for a cloud model.

But that conclusion assumed the small model's job is to **do the work.** A reader pushed back with a better question: what if its job is to **decide who does the work?** Judging a task is cheaper than solving it. So we ran another experiment — same 4 GB laptop, same hidden-grader harness — to find two things: where a small model's limit *actually* is across general tasks, and whether a small model can earn its keep as a **router** instead of a worker.

## The setup

The [harness](/blog/2026/06/13-what-an-ai-agent-harness-actually-does/) is the same one from the coding study: each task runs against a hidden grader, so every result is automated and reproducible. The new task suite is **48 general tasks** — three domains (reasoning, extraction, writing) × four difficulty tiers (easy → extreme) × four tasks each — graded by exact answer, number match, JSON-field match, or a constraint checker (word counts, lipograms, acrostics). We ran **six models × 48 tasks × 3 repetitions = 864 trials**, plus a separate batch of open-ended writing prompts judged by hand.

A quick but important detour: our first run was contaminated by the grader, not the models. Tasks failed because a model answered `$1,240.50` where the grader wanted `1240.50`, or `The Widget Pro` where it wanted `Widget Pro` — *correct extraction, punished for formatting.* That is the same trap the coding study taught us to watch for, and a calibration pass caught it. We normalised currency, commas, and a leading article before comparison. Measure the capability, never the formatting.

## Finding 1: reasoning-tuning beats size

Here is the capability matrix — overall, and broken out by domain:

<Table data={
    {
        headers: ['Model', 'Overall', 'Extraction', 'Reasoning', 'Writing'],
        rows: [
            ['qwen3:4b', '94%', '100%', '94%', '88%'],
            ['qwen3:1.7b', '88%', '98%', '90%', '77%'],
            ['granite3.3:2b', '72%', '94%', '50%', '73%'],
            ['qwen2.5-coder:7b', '72%', '88%', '71%', '56%'],
            ['qwen2.5-coder:3b', '69%', '94%', '54%', '58%'],
            ['qwen2.5-coder:1.5b', '63%', '88%', '50%', '52%'],
        ]
    }
} />

Read the top two rows against the bottom four. **qwen3:1.7b scores 88% and beats qwen2.5-coder:7b at 72% — while being four times smaller.** The coder models, which *won* the coding study, are the *weakest* generalists here. The lesson is blunt: capability is not monotonic in size, and it is not even consistent across model families. A model tuned for reasoning runs circles around a bigger model tuned for code the moment you step off the code.

That detail matters more than it looks, and it comes back to bite us later: **the biggest model in your fleet is not necessarily the best one.**

## Finding 2: the limit is real, and it has a shape

Small models are not uniformly weak — they fail in specific, predictable places:

- **Easy and medium extraction and reasoning are solved.** A 1.5B model pulls fields out of text and does two-step arithmetic at near-100%. This is genuinely usable work.
- **Hard reasoning caps at ~75%** even for the best model. The classic trick — "you just overtook the person in second place; what position are you in?" — still catches them.
- **Lipograms are the single hardest thing we measured.** "Describe the sky without using the letter *e*" sits at 50% across the entire fleet. Writing under a hard mechanical constraint needs planning the models do not have.

So the small model has a real, shaped limit. Which sets up the actual question.

## The pivot: can it route?

If a cheap model cannot reliably *do* hard work, can it reliably *sort* hard work — keep the easy stuff and hand the hard stuff up to a bigger model? We measured three routing signals, from the most idealised to the most deployable.

### Signal 1 — the oracle router (the ceiling)

First, the upper bound. Imagine a perfect router that always sends each task to the *cheapest* model that can solve it. It cannot exist (it would need to know the answer in advance), but it tells you how much routing could *possibly* save. Using model size as a cost proxy:

- **Coverage 98%** — some model in the fleet solves all but one task.
- The oracle router's mean cost is **1.79B params vs 7B for "always use the biggest" — 74% cheaper at identical coverage.**

And the cumulative coverage as you add tiers, cheapest first, is the punchline:

<Table data={
    {
        headers: ['Models available up to', 'Size (B)', 'Cumulative coverage'],
        rows: [
            ['qwen2.5-coder:1.5b', '1.5', '65%'],
            ['qwen3:1.7b', '1.7', '90%'],
            ['granite3.3:2b', '2.0', '94%'],
            ['qwen2.5-coder:3b', '3.0', '94%'],
            ['qwen3:4b', '4.0', '98%'],
            ['qwen2.5-coder:7b', '7.0', '98%'],
        ]
    }
} />

Adding the 1.7B model jumps coverage from 65% to **90%**. Everything above it buys 8 more points, and the 3B and 7B coders buy *nothing* for general work. The whole strategy falls out of one row: **run qwen3:1.7b as the workhorse, escalate the ~10% it can't handle to qwen3:4b, and never load the 7B at all.**

### Signal 2 — self-consistency (proactive, and deployable)

The oracle cheats by peeking at the grader. A real router has no grader at dispatch time — it only sees the model's own output. So here is a signal it *can* use: **sample the small model a few times and watch whether it agrees with itself.** Confident, stable answers are accepted; answers that wobble across samples get escalated.

The appeal is that it sidesteps the obvious failure mode of asking a model "is this hard?" — a weak model is exactly the one that *can't tell* a hard question is hard, and will cheerfully label it easy. Instability is harder to fake than a self-rating. We sampled qwen3:1.7b five times per task and scored two definitions of "agree":

<Table data={
    {
        headers: ['Agreement defined as', 'Accept precision', 'Escalation rate', 'Missed errors'],
        rows: [
            ['Same text (any wording differs = disagree)', '100%', '60%', '0%'],
            ['Same answer (8 = "8 days" = "the answer is 8")', '97%', '38%', '2%'],
        ]
    }
} />

"Accept precision" is the chance an accepted answer is actually right; "missed errors" are confident-but-wrong answers that slip through. The first row is remarkable: **when this model agrees with itself word-for-word, it is right every single time** — zero bad accepts. The catch is it over-escalates, because it counts "8" and "8 days" as a disagreement. Comparing the *answer* instead of the *wording* cuts wasted escalation from 60% down to 38%, at the cost of exactly one confident-wrong slip. That is not a bug to fix — it is a **dial.** Turn it toward text-agreement when mistakes are expensive, toward answer-agreement when compute is.

### Signal 3 — feedback escalation, and the corpus it leaves behind

The first two signals act before the user sees anything. The third one is the human in the loop: a **thumbs-down re-routes the same question to a stronger model**, and — just as importantly — every disliked question is saved to a deduplicated corpus for later.

Here is where Finding 1 comes back to bite. The naive way to "escalate to a stronger model" is to pick a *bigger* one — and that is wrong. Escalating qwen3:1.7b's failures by size would send them to granite or the 7B coder, both of which are *worse* at general tasks. Escalation has to follow **measured capability, not parameter count.** Routed correctly to qwen3:4b, the bigger model recovered 2 of the 5 disliked answers.

The other 3 are the interesting part. They were the marble-fraction trick and **both lipograms** — the exact hard tail from Finding 2, the tasks *no* tier in the fleet can solve. Escalation can't fix those, and that is precisely why the corpus matters: it automatically collects the questions that routing alone will never rescue. That pile is your eval set, your fine-tuning data, and your prompt-engineering to-do list — the questions worth a human's attention, surfaced for free.

## A note on the writing

The open-ended writing prompts told the same story from a different angle. Every model, down to 1.5B, writes **fluent** prose. What separates them is **discipline** — obeying the instruction. The small coders wrote four lines when asked for a two-line rhyme, three sentences when asked for two. Only two of six models produced an actual rhyme. The best writer, qwen3:4b, also had the worst obedience, cheerfully appending an emoji checklist to a "short email." Fluency is free at 1.5B; *following the constraint* is the thing that scales with reasoning-tuning — the same axis the whole routing argument rides on.

## The takeaway

The small model's ceiling is real, but the framing was wrong. Asked to be the worker, a 4 GB model is a mediocre generalist. Asked to be the **dispatcher**, it is excellent — and the architecture writes itself:

- **qwen3:1.7b as the always-on workhorse**, covering ~90% of general traffic on hardware you already own.
- **Self-consistency as the escalator** — when the small model disagrees with itself, hand the task up. A tunable dial between cost and safety.
- **Escalate by measured capability, not size** — and capture every thumbs-down into a hard-case corpus, because that corpus is where the next round of real improvement comes from.

Stop asking the small model to do the work. Ask it to route the work, and it will save you most of your compute while quietly building the dataset that makes everything downstream better.

*This closes the small-model thread of the series — from [which agent wins](/blog/2026/06/13-the-coding-agent-shootout-and-the-best-harness-we-could-not-read/), to [what a tool call really is](/blog/2026/06/15-what-a-native-tool-call-actually-is/), to the 4 GB model's real job. Next, we turn the lens back on the big agents.*


01468f0733b58c321602a62d2f81c9448ea48df1c88cc4b2d188d3a423056e5a

An empirical study of small local models across reasoning, extraction and writing on a 4 GB RTX 3050, plus three routing signals — oracle upper bound, self-consistency, and feedback escalation — that turn a small model into a dispatcher.

We pushed a 4 GB laptop model past coding into reasoning, extraction and writing to find its limit — then measured a better use for it entirely: not doing the work, but routing it. A reasoning-tuned 1.7B model covers 90% of a general workload, and a perfect router is 74% cheaper at equal quality.

The Small Model's Real Job Is Dispatch, Not Work

{/* banner prompt: High-key cinematic shot in a vast, minimalist white space with atmospheric haze; a single, compact, pristine cube on the floor projects a focused, coherent beam of light that forms a perfect code structure on a far wall, while much larger, ghostly, and fragmented machines dissolve into the bright background, 16:9, volumetric lighting, sense of scale and clarity. */}

The first five posts in this series were the telescope and the microscope — what agents [are](/blog/2026/06/13-codex-pi-and-hermes-three-agents-three-species/), what a [harness does](/blog/2026/06/13-what-an-ai-agent-harness-actually-does/), how they [learn](/blog/2026/06/13-skills-and-memory-how-ai-agents-actually-learn/), which one [wins](/blog/2026/06/13-the-coding-agent-shootout-and-the-best-harness-we-could-not-read/), and how one [actually writes a line of code](/blog/2026/06/13-how-a-coding-agent-actually-writes-code/). All of it was *reading* — code and behaviour. This post is different: we stopped reading and **ran the experiment.**

The question: on cheap, local hardware — a 4GB laptop GPU — **what actually makes a small model a good coding agent?** Not "which frontier model wins" (you can't run those at home), but: given a model that fits in 4 GB, which *harness choices* matter? Edit format? Retry? Task decomposition? A planner model? We built a measurement harness, wrote a suite of coding tasks with automated pass/fail oracles, and ran roughly **900 trials**. The results overturned most of our intuitions.

## The setup

A gaming laptop with an **RTX 3050 (4 GB VRAM)** serving models via Ollama; a separate machine running the agent and holding the code. The harness runs each task in a fresh copy of its workspace, lets the model attempt it, then grades it with a hidden `pytest` oracle — the test is the only judge, so every result is automated and reproducible.

The **task suite** is 28 self-contained coding tasks: add a function, fix a bug, small refactors, implement-from-spec, multi-file changes, and a deliberately hard tier (a bowling scorer, a circular buffer, rational-number arithmetic, a 24-hour clock, a book-store discount optimiser, variable-length-quantity encoding). Each oracle was checked two ways — it must *fail* on the starting code and *pass* on a correct solution. Calibrated this way, the simplest configuration solves about 71%, leaving real room to measure whether the fancy techniques help.

We varied **one lever at a time** around that baseline (`qwen2.5-coder:3b`, whole-file edits, single-shot), ran each task/config combination three times to smooth out the noise of a non-deterministic model, and recorded success rate, tokens, and wall-time.

## The results

<Table data={
    {
        headers: ['Config', 'Success', 'Sec/run', 'What changed vs baseline'],
        rows: [
            ['baseline (coder-3b, whole-file)', '71%', '2.6', 'reference'],
            ['retry (run test, feed failure back)', '70%', '8.6', 'no change'],
            ['decomposition (plan then execute)', '64%', '5.3', 'hurt'],
            ['role-split (planner + editor)', '66%', '134', 'hurt + 50x slower'],
            ['model: qwen3:1.7b', '43%', '50', 'much worse coder'],
            ['model: granite3.3:2b', '39%', '2.2', 'much worse coder'],
            ['edit format: diff', '24%', '5.9', 'far worse'],
            ['edit format: native tool-calls', '0%', '2.1', 'impossible'],
        ]
    }
} />

Read that table for a minute, because almost every row is a surprise.

## Finding 1: the edit format is the dominant lever

How the model is asked to express a change matters more than anything else. Whole-file rewrite: **71%**. Search/replace diff: **24%**. Native function-calling: **0%**.

That **0%** deserves a pause. We gave the model proper OpenAI-style tools and asked it to call them — and across 84 attempts it never once produced a usable tool call. It *knew what to do*: it emitted perfectly correct JSON like `{"name": "read_file", "arguments": {"path": "config.py"}}` — but as a markdown code block in its **message text**, never in the structured `tool_calls` field the protocol requires. A native-tool agent (Codex, Pi, anything function-calling-based) sees an empty tool-call list and does nothing. **The small coder model literally cannot drive a native-tool agent.**

Diff (24%) is a softer version of the same lesson: the model produces well-formed search/replace blocks, but on longer files the "search" text drifts from the source by a character and the edit won't apply. Whole-file editing sidesteps both problems — there's nothing to match, nothing to format. For a small model, **make the easy thing the only thing.**

## Finding 2: "best coder" is not "best tool-caller"

We tried three models that fit in 4 GB. The coding-specialised `qwen2.5-coder:3b` (71%) crushed the newer general models `qwen3:1.7b` (43%) and `granite3.3:2b` (39%) at producing correct edits.

Here's the twist: **qwen3:1.7b is the *better tool-caller*** — it can emit native `tool_calls` and drive an agent loop where the coder model can't — yet it's the *worse coder*. Tool-calling ability and coding ability are different axes, and a model can be strong on one and weak on the other. If your harness leans on native tool-calling, you're implicitly selecting for the tool-caller and possibly against the coder. (qwen3:1.7b is also strikingly slow and verbose here — ~50 s/run, ten times the output tokens — despite being told not to "think.")

## Finding 3: the clever agentic scaffolding didn't help — and mostly hurt

This is the one that stings, because it's the stuff everyone reaches for.

- **Retry** (run the test, feed the failure back, let it fix): **neutral** (70% vs 71%). Retry recovers *near-misses*. But this suite's failures are the hard tier — bowling, VLQ — which are *capability-bound*: the 3B can't write a correct bowling scorer no matter how many times you show it the failing test. **Retry can't buy capability the model doesn't have.**
- **Decomposition** (plan the task into steps, then execute): **−7 points**. The planning step adds tokens and structure a small model handles poorly; a slightly wrong plan actively misleads it.
- **Role-split** (a stronger `qwen3:4b` planner directing the coder): **−5 points and ~50× slower**. On 4 GB the two models can't co-reside, so the GPU thrashes swapping them in and out. A separate planner didn't help and cost two minutes a task.
- **Stacking** the "improvements" together regressed *below* baseline, because the negative levers compound.

The pattern: orchestration tricks that help a frontier model coordinate **do not rescue a small model whose failures are a capability ceiling, and several make it worse** by adding surface area for a limited model to trip over.

## The follow-up: can we craft tool-calls the small model *can* do?

Finding 1 left a loose end. Native tool-calling scored 0% only because the model emits tool intent as *text*, not in the `tool_calls` field. So we asked: what if we meet the model where it is — build our own tool-call protocol that parses the JSON-in-content it naturally produces — and give it a real agent loop (read → edit → run → fix)?

We did. A lenient parser (accepting fenced JSON, `<tool_call>` tags, or bare JSON), a small tool set, and a loop that hides the grading test until the end. Two results:

1. **A custom text protocol rescues tool-calling: 0% → ~56%.** Meeting the model where it is *works* where the rigid native protocol is impossible. The 3B genuinely drives a multi-step loop.
2. **But the loop still loses to single-shot — and its value depends on the model.** On the strong coder it *hurt* (71% single-shot → 56% in the loop: more turns, more chances to go wrong). On the weaker-but-tool-capable qwen3:1.7b it *helped* (43% → 58%). Neither beat plain coder-3b doing one whole-file edit.

So you *can* engineer a small model into being an agent at all — and that's a genuinely useful capability — but the engineering doesn't beat the simplest possible setup.

## What it adds up to

For a small model on scoped coding tasks, the high-leverage choices are the **editor model** and the **edit format** — not the agentic scaffolding. The recommended 4 GB configuration is almost embarrassingly plain: **`qwen2.5-coder:3b`, whole-file edits, single-shot.** 71% at under three seconds a task. Everything fancier was neutral, slower, or worse.

This both confirms and sharpens the series' running thesis. We've said all along that *the harness, not the model, is where the engineering lives* — and that's still true: the harness's edit-format and model choices made the difference between 0% and 71%. But there's a hard limit the experiment made vivid. A harness can fix problems of **interface** — give the model an edit format and a tool-call protocol it can actually use — and the payoff is enormous. It cannot fix problems of **capability** — when the model simply can't write the bowling scorer, no amount of retry, planning, or multi-model orchestration conjures the answer, and the cleverer the scaffolding, the more often it backfires. With a small model you feel that ceiling directly, and the winning move is to stop adding machinery and get out of the model's way.

There's a nice corollary for anyone running local AI on modest hardware: you don't need an elaborate agent framework to get real value from a 4 GB GPU. You need the right small coder model, whole-file editing, and the discipline to keep it simple.

*The harness for this study — the task suite, the levers, the custom tool-call loop, and the full results — is open source. This experiment grew out of trying to give a real coding agent a local brain; the surprise was how little the brain needed around it.*

0c62a9ca5856404573db16db808a2aa19791643978e8def22253ec0825fcba55

An empirical study of small-model coding agents on a 4 GB RTX 3050: edit format, model choice, retry, decomposition, role-split, and a custom text tool-call loop, measured across ~900 automated trials.

We stopped theorizing and ran ~900 trials on a 4 GB laptop GPU to measure what makes a small local model a good coding agent. The result overturns intuition: the edit format and the editor model decide almost everything, the clever agentic scaffolding mostly hurts, and the simplest setup won.

Tag: Local Llm

Tag: Local Llm

The Model That Aced Olympiad Math Can't Write a Tweet

The 4 GB Card You Already Own Can Reason Now

The Small Model's Real Job Is Dispatch, Not Work

The 4 GB Experiment: What Actually Makes a Small Model a Good Coding Agent